DL2: A Deep Learning-Driven Scheduler for Deep Learning Clusters

Efficient resource scheduling is essential for maximal utilization of expensive deep learning (DL) clusters. Existing cluster schedulers either are agnostic to machine (ML) workload characteristics, or use heuristics based on operators' understanding particular ML framework and workload, which less efficient not general enough. In this article, we show that DL techniques can be adopted design a generic scheduler. Specifically, propose DL2, DL-driven scheduler clusters, targeting global training job expedition by dynamically resizing resources allocated jobs. DL2 advocates joint supervised reinforcement approach: neural network warmed up via offline traces produced the existing scheduler; then plugged into live cluster, fine-tuned carried out throughout progress jobs, used deciding allocation in an online fashion. We implement Kubernetes enable dynamic scaling jobs MXNet. Extensive evaluation shows outperforms fairness (i.e., DRF) 44.1 percent expert heuristic Optimus) 17.5 terms average completion time.